Multi-device communication management

ABSTRACT

A device implementing the subject technology may include at least one processor configured to transmit an allocation request requesting allocation of a group communication session with a plurality of devices and receive an allocation response in response to the allocation request, the allocation response including credential information for the device to use to join the group communication session. The at least one processor may be further configured to transmit an allocation bind request with the credential information to join the group communication session using the credential information and receive an allocation bind success response in response to the allocation bind request, the allocation bind success response indicating that the device has joined the group communication session. The at least one processor may be further configured to provide a join notification to the plurality of devices via an intermediary device to notify that the device has joined the group communication session.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims the benefit of U.S. Provisional PatentApplication Ser. No. 62/565,910, entitled “Multi-Device CommunicationManagement,” filed on Sep. 29, 2017, which is hereby incorporated byreference in its entirety for all purposes.

TECHNICAL FIELD

The present description relates generally to communication managementbetween multiple devices, including management of a group communicationsession between multiple devices.

BACKGROUND

Communication between multiple devices (e.g., multi-devicecommunication) has been utilized more frequently over time. For example,multiple users may use their respective devices to participate in agroup communication session to communicate with one another. Withadvancement of technologies, a range of tasks that may be performed bymultiple devices communicating with each other has expanded over time.For example, multi-device communication may enable users to performvarious tasks such as participating in an audio/video conference,sharing files, streaming media to another device, etc.

BRIEF DESCRIPTION OF THE DRAWINGS

Certain features of the subject technology are set forth in the appendedclaims. However, for purpose of explanation, several embodiments of thesubject technology are set forth in the following figures.

FIG. 1 illustrates an example network environment in which amulti-device communication management system may be implemented inaccordance with one or more implementations.

FIG. 2 illustrates an example electronic device that may implement amulti-device communication management system in accordance with one ormore implementations.

FIG. 3 illustrates an example server that may implement a multi-devicecommunication management system in accordance with one or moreimplementations.

FIG. 4 illustrates an example server arranged in a distributedarchitecture that may implement a multi-device communication managementsystem in accordance with one or more implementations.

FIG. 5 illustrates a flow diagram of an example process of amulti-device communication management system in accordance with one ormore implementations.

FIGS. 6-14 illustrate flow diagrams of example processes of amulti-device communication management system in accordance with one ormore implementations.

FIG. 15 illustrates a flow diagram of an example process of amulti-device communication management system in accordance with one ormore implementations.

FIG. 16 illustrates a flow diagram of an example process of amulti-device communication management system in accordance with one ormore implementations.

FIGS. 17 and 18 illustrate flow diagrams of example processes of amulti-device communication management system in accordance with one ormore implementations.

FIGS. 19-23 illustrate flow diagrams of example processes of amulti-device communication management system in accordance with one ormore implementations.

FIGS. 24-27 illustrate flow diagrams of example processes of amulti-device communication management system in accordance with one ormore implementations.

FIG. 28 illustrates a flow diagram of an example process of amulti-device communication management system in accordance with one ormore implementations.

FIG. 29 illustrates a flow diagram of an example process of amulti-device communication management system in accordance with one ormore implementations.

FIG. 30 illustrates a flow diagram of an example process of amulti-device communication management system in accordance with one ormore implementations.

FIG. 31 illustrates an example electronic system with which aspects ofthe subject technology may be implemented in accordance with one or moreimplementations.

DETAILED DESCRIPTION

The detailed description set forth below is intended as a description ofvarious configurations of the subject technology and is not intended torepresent the only configurations in which the subject technology can bepracticed. The appended drawings are incorporated herein and constitutea part of the detailed description. The detailed description includesspecific details for the purpose of providing a thorough understandingof the subject technology. However, the subject technology is notlimited to the specific details set forth herein and can be practicedusing one or more implementations. In one or more implementations,structures and components are shown in block diagram form in order toavoid obscuring the concepts of the subj ect technology.

With the advancement of device and communication technologies, groupcommunication sessions may frequently include more than two devices. Forexample, an audio/video conference among a large number of people usingtheir respective devices has become possible. However, communicationbetween more than two devices may face challenges with respect toproviding an optimal user experience to the user of each device. Forexample, poor communication conditions at one device may cause a delayand/or a suboptimal user experience at one or more of the other devices.Furthermore, when one user experiences more audio delay during anaudio/video conference than one or more of the other users, it may bedifficult for the user to participate in the audio/video conference. Forexample, by the time the user who experiences more audio delay hears acommunication, another user (who received the communication with lessdelay) may have already started responding, and therefore the user whoexperiences more audio delay may not be able to effectively participatein the audio/video conference.

The subject system for multi-device communication management providessolutions to these and other problems by jointly managing/coordinatingmultiple different devices in a group communication session, whileallowing a text based group communication session to be switched to anaudio/video group communication session with minimal latency. Forexample, the subject system jointly coordinates the quality of the audioand/or video streams provided by the devices in the group communicationsession such that each device can access high quality audio and/or videostreams from other devices while ensuring that any devices withbandwidth constraints have access to lower quality streams. Furthermore,the subject system jointly controls the audio delay experienced by theusers in the group communication session such that no single userexperiences significantly more audio delay than any other user. Inaddition, the subject system allows users to transition between devicesduring a group communication session, such that the users can seamlesslyparticipate in the group communication session while switching betweendifferent devices.

FIG. 1 illustrates an example network environment 100 in which amulti-device communication management system may be implemented inaccordance with one or more implementations. Not all of the depictedcomponents may be used in all implementations, however, and one or moreimplementations may include additional or different components thanthose shown in the figure. Variations in the arrangement and type of thecomponents may be made without departing from the spirit or scope of theclaims as set forth herein. Additional components, different components,or fewer components may be provided.

The network environment 100 includes one or more electronic devices102A-D and one or more servers 110, such as a cloud of servers, whichfor explanatory purposes may be collectively referred to as a server110. The electronic devices 102A-D may be communicatively coupled to theserver 110 by the communication links 122A-D, respectively. In one ormore implementations, one or more of the communication links 122A-D mayinclude, and/or may be communicatively coupled to, one or more wired orwireless network components, such as routers, switches, access points,base stations, and the like. In one or more implementations, theelectronic devices 102A-D may communicate with each other directlyand/or via the server 110, such as in a group communication session.

The electronic devices 102A-D may be, for example, portable computingdevices such as laptop/desktop computers, smartphones, tablet devices,wearable devices (e.g., watches, bands, etc.), or other appropriatedevices that include one or more wired or wireless interfaces, such asone or more near field communication (NFC) radios, WLAN radios,Bluetooth radios, Zigbee radios, cellular radios, and/or other wirelessradios. In FIG. 1, by way of example, the electronic device 102A isdepicted as a tablet device, the electronic device 102B is depicted as alaptop device, the electronic device 102C is depicted as a mobiledevice, and the electronic device 102D is depicted as a desktopcomputer. One or more of the electronic devices 102A-D may be, and/ormay include all or part of, the electronic device discussed below withrespect to FIG. 2 and/or the electronic system discussed below withrespect to FIG. 31. The server 110 may be, and/or may include all orpart of, the server discussed below with respect to FIGS. 3 and 4 and/orthe electronic system discussed below with respect to FIG. 31.

In one or more implementations, the electronic device 102A and theelectronic device 102D may be registered to a same user account, such asthrough the server 110. Thus, a user associated with the user accountmay receive communications, e.g., messages, phone calls, notifications,etc., via one or both of the electronic devices 102A,D. In one or moreimplementations, the electronic devices 102A,D may also be connected viaa local peer-to-peer connection, such as BTLE, when the electronicdevices 102A,D are located proximal to one another, such as within theBTLE transmission range. The user may also use one or more of theelectronic devices 102A,D, such as the electronic device 102A, in agroup communication session with one or more other devices, such as theelectronic devices 102B-C, which may each be associated with a differentuser account. In a group communication session, audio and/or videocaptured at the electronic device 102A may be transmitted to the otherelectronic devices 102B-C, e.g., via the server 110. Similarly, theelectronic device 102A may receive audio and/or video streamscorresponding to the other electronic devices 102B-C via the server 110.

In the subject system, the server 110 may facilitate securely forming agroup communication session between one or more of the electronicdevices 102A-D, such as an audio/video conference, and/or securelytransitioning between different communication modalities (e.g.,messaging, audio, or video) for a given group communication session. Forexample, the users of one or more of the electronic devices 102A-D maybe participating in a group messaging exchange and may decide that theywould like to transition from the group messaging exchange to anaudio/video conference. The server 110 may facilitate transitioning thegroup communication session from the group messaging exchange to theaudio/video conference. An example process for transitioning betweendifferent communication modalities for a given group communicationsession is discussed further below with respect to FIG. 30. In one ormore implementations, the server 110 may also facilitate securely addingone or more of the electronic devices 102A-D to an existing groupcommunication session. Example processes of a server 110 securelyforming a group communication session and/or securely adding one or moreof the electronic devices 102A-D to an existing group communicationsession are discussed further below with respect to FIGS. 6-13 and19-27.

In the group communication session, such as an audio/video conference,the electronic devices 102A-D may provide independent audio and/or videostreams to each other. However, since the uplink/downlinkbandwidth/network conditions of the communication links 122A-D of theelectronic devices 102A-D may differ, the electronic devices 102A-Dand/or the server 110 may coordinate the quality of the audio and/orvideo streams provided and/or made available by the electronic devices102A-C for the group communication session to ensure that sufficientlylow bit rate streams are available for one or more of the electronicdevices 102A-C that have a communication link that can only support lowbit rate streams while still allowing others of the electronic devices102A-C to receive high bit rate streams, e.g., when their respectivecommunication links can support high bit rate streams. Example processesof the server 110 managing the quality of the audio/video streamsprovided and received by the electronic devices 102A-D in a groupcommunication session are discussed further below with respect to FIGS.16 and 29.

In one or more implementations, variations in network conditions of thecommunications links 122A-D may result in different amounts of jitteroccurring at one or more of the electronic devices 102A-D. The differentamounts of jitter may result in different jitter buffer levels andtherefore different audio output delays across the electronic devices102A-D. In the subject system, the server 110 may coordinate the audiooutput delay resulting from the jitter buffer level variations at theelectronic devices 102A-D to ensure that the audio output delay issubstantially uniform across the electronic devices 102A-D, while alsoensuring that the jitter buffers of the electronic devices 102A-D do notexperience buffer overruns or underruns. Example processes of the server110 coordinating the audio output delay across the electronic devices102A-D is discussed further below with respect to FIGS. 15 and 28.

The server 110 may also facilitate allowing a user to seamlessly switchelectronic devices 102A,D while participating in a group communicationsession. For example, the user may initially use the electronic device102A to participate in a group communication session (or otherwiseperform a task), and may switch to the electronic device 102D (oranother electronic device registered to the account of the user) tocontinue participating in the group communication session (or continueperforming the task). Thus, when a user is participating in a groupcommunication session (or performing another task) using the electronicdevice 102A, the server 110 may coordinate preparing, or warming up, oneor more other electronic devices 102D registered to the account of theuser, such that the group communication session (and/or other task) maybe seamlessly and substantially instantly handed off to the one or moreother electronic devices 102D, e.g., upon receiving a user requesttherefor. Example processes of the server 110 coordinating thepreparation of electronic devices 102D for a seamless handoff of a groupcommunication session (and/or other task) are discussed further belowwith respect to FIGS. 5, 17, and 18.

In one or more implementations, the subject system may allow a digitalassistant to participate in, and/or assist, in a group communicationsession, such as an audio/video conference, a group messaging session,and the like. For example, a user of one of the electronic devices102A-C may request, e.g. verbally, that the digital assistant schedule areminder for the participants in the call to discuss the topic againnext week, to schedule a lunch with the participants in the call,share/email a document, and/or any other task. The digital assistant,which may be implemented by the server 110 and/or by one or more of theelectronic devices 102A-C, may then schedule the reminder, lunch, etc.for each of the user accounts associated with the electronic devices102A-C participating in the group communication session.

When the digital assistant is implemented centrally, such as by theserver 110, the server 110 may receive packets from the electronicdevices 102A-C that correspond to digital assistant activities/requestsinitiated on the respective electronic devices 102A-C. The server 110may process the digital assistant activities/requests and may broadcastresultant activities, such as scheduling a lunch, to each of theelectronic devices 102A-C.

In one or more implementations, a digital assistant on each of theelectronic devices 102A-C may be maintain a context and/or stateassociated with the group communication session during the groupcommunication session. If, for example, the user of the electronicdevice 102A switches to the electronic device 102D during the groupcommunication session, the context of the group communication sessionmaintained by the digital assistant on the electronic device 102A may behanded off to the digital assistant on the electronic device 102D. Inthis manner, the interactions/history with the digital assistant on theelectronic device 102A can be continued on the electronic device 102D.

In one or more implementations, the subject system may allow the server110 to aggregate the video, audio, and or messaging communications for agiven group communication session, and generate a summary of the groupcommunication session, that may subsequently be provided to, and/or madeavailable to, one or more of the electronic devices 102A-C and/or otherelectronic devices. The summary may be, for example, a video summary, atranscript, a podcast, or the like. The summary may include the entiretyof the group communication session, and/or may be condensed, sped-up, orotherwise processed version of the group communication session.

FIG. 2 illustrates an example electronic device 102A that may implementthe multi-device communication management system in accordance with oneor more implementations. Not all of the depicted components may be usedin all implementations, however, and one or more implementations mayinclude additional or different components than those shown in thefigure. Variations in the arrangement and type of the components may bemade without departing from the spirit or scope of the claims as setforth herein. Additional components, different components, or fewercomponents may be provided.

The electronic device 102A may include, among other components, a hostprocessor 202, a memory 204, and a communication interface 206. The hostprocessor 202, which may also be referred to as an application processoror a processor, may include suitable logic, circuitry, and/or code thatenable processing data and/or controlling operations of the electronicdevice 102A. In this regard, the host processor 202 may be enabled toprovide control signals to various other components of the electronicdevice 102A.

The host processor 202 may also control transfers of data betweenvarious portions of the electronic device 102A. Additionally, the hostprocessor 202 may enable implementation of an operating system orotherwise execute code to manage operations of the electronic device102A. The memory 204 may include suitable logic, circuitry, and/or codethat enable storage of various types of information such as receiveddata, generated data, code, and/or configuration information. The memory204 may include, for example, random access memory (RAM), read-onlymemory (ROM), flash, and/or magnetic storage.

The communication interface 206 may be used by the host processor 202 tocommunicate via one or more communication protocols, such as Wi-Fi,cellular, Ethernet, Bluetooth, Zigbee, or NFC, or the like. In one ormore implementations, the communication interface 206 may be, mayinclude, and/or may be communicatively coupled to a first radiofrequency (RF) circuit, such as a WLAN circuit, a cellular RF circuit,and/or a second RF circuit, such as a Bluetooth circuit and/or an NFCcircuit, or the like.

In one or more implementations, one or more of the host processor 202,the memory 204, the communication interface 206, and/or one or moreportions thereof, may be implemented in software (e.g., subroutines andcode), hardware (e.g., an Application Specific Integrated Circuit(ASIC), a Field Programmable Gate Array (FPGA), a Programmable LogicDevice (PLD), a controller, a state machine, gated logic, discretehardware components, or any other suitable devices) and/or a combinationof both.

FIG. 3 illustrates an example server 110 that may implement themulti-device communication management system in accordance with one ormore implementations. Not all of the depicted components may be used inall implementations, however, and one or more implementations mayinclude additional or different components than those shown in thefigure. Variations in the arrangement and type of the components may bemade without departing from the spirit or scope of the claims as setforth herein. Additional components, different components, or fewercomponents may be provided.

The server 110 may include, among other components, a processor 302, amemory 304, and a communication interface 306. The processor 302 mayinclude suitable logic, circuitry, and/or code that enable processingdata and/or controlling operations of the server 110. In this regard,the processor 302 may be enabled to provide control signals to variousother components of the server 110, and/or other servers/devicescommunicatively coupled thereto.

The processor 302 may also control transfers of data between variousportions of the server 110. Additionally, the processor 302 may enableimplementation of an operating system or otherwise execute code tomanage operations, such as server-side operations, of the server 110.The memory 304 may include suitable logic, circuitry, and/or code thatenable storage of various types of information such as received data,generated data, code, and/or configuration information. The memory 304may include, for example, RAM, ROM, flash, and/or magnetic storage.

The communication interface 306 may be used by the processor 302 tocommunicate via a communication protocol, such as Wi-Fi, cellular,Ethernet, Bluetooth, Zigbee, or NFC, or the like. In one or moreimplementations, the communication interface 306 may be, may include,and/or may be communicatively coupled to a first radio frequency (RF)circuit, such as a WLAN circuit, a cellular RF circuit, and/or a secondRF circuit, such as a Bluetooth circuit and/or an NFC circuit, or thelike.

In one or more implementations, one or more of the processor 302, thememory 304, the communication interface 306, and/or one or more portionsthereof, may be implemented in software (e.g., subroutines and code),hardware (e.g., an ASIC, an FPGA, a PLD, a controller, a state machine,gated logic, discrete hardware components, or any other suitabledevices) and/or a combination of both.

FIG. 4 illustrates an example server 110 arranged in a distributedarchitecture that may implement a multi-device communication managementsystem in accordance with one or more implementations. Not all of thedepicted components may be used in all implementations, however, and oneor more implementations may include additional or different componentsthan those shown in the figure. Variations in the arrangement and typeof the components may be made without departing from the spirit or scopeof the claims as set forth herein. Additional components, differentcomponents, or fewer components may be provided.

The server 110 may include multiple servers 412, 414, 416 that may becommunicatively coupled to one another. The multiple servers 412, 414,416 may include a relay server 412, an allocator server 414, and anotification server 416. One or more of the servers 412, 414, 416 may beconfigured to communicate with one or more of the electronic devices102A-D, such as to facilitate a group communication session. In one ormore implementations, one or more of the servers 412, 414, 416 may notbe communicatively coupled to one or more others of the servers 412,414, 416.

FIG. 5 illustrates a flow diagram of an example process 500 of themulti-device communication management system in accordance with one ormore implementations. For explanatory purposes, the process 500 isprimarily described herein with reference to the electronic devices102A,D and the server 110 of FIG. 1. However, the process 500 is notlimited to the electronic devices 102A,D and/or the server 110. Theelectronic devices 102A,D and the server 110 are also presented asexemplary devices and the operations described herein may be performedby any suitable devices. Further for explanatory purposes, the blocks ofthe process 500 are described herein as occurring in serial, orlinearly. However, multiple blocks of the process 500 may occur inparallel. In addition, the blocks of the process 500 need not beperformed in the order shown and/or one or more of the blocks of theprocess 500 need not be performed and/or can be replaced by otheroperations. In one or more implementations, one or more of theoperations described as being performed at the server 110 may beperformed at least in part at one or more of the electronic devices102A,D, and vice-versa. In one or more implementations, one or more ofthe communications between the electronic devices 102A,D may becommunicated via a peer-to-peer connection and/or one or more of thecommunications between the electronic devices 102A,D may be relayedthrough the server 110, such as over a network.

In the process 500, the electronic device 102A starts performing a task(512). For example, a user of the electronic device 102A may initiatethe task on the electronic device 102A (e.g., audio/video conference,audio streaming, document presenting, voice call, etc.). In one or moreimplementations, the task may be performed in conjunction with theserver 110, such as participating in a group communication session,and/or the electronic device 102A may notify the server 110 that it hasinitiated, and/or is engaging in, the task. In one or moreimplementations, the task may be a task that the user may wish, at somepoint, to handoff from one electronic device 102A to another electronicdevice 102D registered to the account of the user, such when the user isparticipating in an audio/video conference.

Device capabilities of electronic devices and/or user attentioninformation may indicate that a particular electronic device is capableof having a particular task handed off to it, and/or that a user isintending to handoff a task to a particular electronic device. Forexample, if a user faces the electronic device 102D, instead of theelectronic device 102A, the attention of the user on the electronicdevice 102D may be an indication that the user would like to handoff thetask to the electronic device 102D. The direction the user is facing maybe determinable, for example, from an image captured from a cameradevice, such as a camera device of the electronic device 102A, theelectronic device 102D, and/or some other device. In another example,the user may wish to change devices based on the respective capabilitiesof the devices, e.g. the electronic device 102D may have a larger screenthan the electronic device 102A. Thus, the server 110 (and/or theelectronic device 102A) may obtain information about device capabilitiescorresponding to the electronic devices 102A,D, and/or user attentioninformation corresponding to the electronic devices 102A,D (514), and/orany other electronic devices that are registered to the same useraccount as the electronic device 102A.

In one or more implementations, the server 110 may determine, based atleast in part on the received information, a likelihood that the userwill request to handoff the task from the electronic device 102A to theelectronic device 102D, or another of the electronic devices associatedwith the account of the user. If the server 110 determines that there isa high likelihood that the user may request to handoff the electronicdevice 102D and/or to another electronic device, the server 110 maytransmit a task handoff preparation request (515) to the electronicdevice 102D indicating that the electronic device 102D should prepare toreceive a possible handoff of the task. Thus, the task handoffpreparation request may be received by the electronic device 102D beforea request to handoff the task has actually been received.

For example, the server 110 may determine that there is a highlikelihood that the user may request to handoff the task to theelectronic device 102D when the user is located with a close proximityof the electronic device 102D. The proximity of the user to theelectronic device 102D may be determinable from known locations of theelectronic devices 102A,D (e.g., from positioning systems), byidentifying the user in images and/or audio captured by the electronicdevice 102D, by determining that the electronic device 102D is able toreceive short range wireless signals from the electronic device 102A,e.g. NFC and/or Bluetooth, and/or by any other positioning mechanisms.Alternatively, and/or in addition, the server 110 may determine thatthere is a high likelihood that the user may request to handoff the taskwhen the user begins interacting with the electronic device 102D whileengaging in the task on the electronic device 102A, such as when theuser opens, unlocks, and/or logs into the electronic device 102D, and/orconversely when the user closes, locks, and/or logs out of theelectronic device 102A.

In one or more implementations, when the electronic device 102D receivesthe task handoff preparation request from the server 110, the electronicdevice 102D may perform one or more preliminary operations to prepare toreceive a handoff of the task. The preliminary operations may include,for example, the allocation/join processes described above with respectto FIGS. 6-13, exiting a low power mode, powering on one or moretransceivers, connection establishment, e.g. for an audio and/or videoconference/call, connecting to a network with a particular quality ofservice and/or bandwidth, powering on audio and/or videoencoders/decoders, powering on/preparing one or more audio and/or videooutput devices, launching an application associated with the task,reserving processor and/or memory resources for performing the task, andthe like. For example, if the electronic device 102D is engaged in abackground operation that requires a significant amount of processing,memory, or network bandwidth, such as downloading an update, or thelike, the electronic device 102D may pause the background operation inanticipation of receiving a handoff of the task.

In one or more implementations, the server 110 may coordinate one ormore security operations to prepare the electronic device 102D toreceive a handoff of the task. For example, the task may requireauthenticating with the server 110 and/or another server, and/orreceiving one or more security keys for participating in the task, suchas keys for joining/participating in an audio/video conference. Thus,the server 110 may coordinate obtaining the appropriate security keysand/or the performing the appropriate authentication protocols for theelectronic device 102D and/or any other electronic devices registered tothe account of the user.

The server 110 may provide the electronic device 102A with informationregarding which other electronic devices the task can be handed off to,such as the electronic device 102D. The electronic devices that the taskcan be handed off to may include, for example, other devices registeredto the account of the user. When the user wishes to handoff the task toanother electronic device, such as the electronic device 102D, theelectronic device 102A may receive user input indicating a request tohandoff the task (516). For example, the electronic device 102A maydisplay to the user a list of the electronic devices that the task canbe handed off to, e.g. as indicated by the server 110, and the user mayselect one of the electronic devices, such as the electronic device102D, to initiate handing off the task.

After receiving the request to handoff the task, the electronic device102A transmits a task handoff request to the electronic device 102D thatrequests that the electronic device 102D prepare to receive the taskbeing performed at the electronic device 102A (518). When the taskhandoff request is received by the electronic device 102D, theelectronic device 102D prepares to receive the task being performed bythe electronic device 102A (520). In one or more implementations, whenthe server 110 has already transmitted the task handoff preparationrequest to the electronic device 102D (515), the electronic device 102Dmay already be prepared and/or substantially prepared to receive thehandoff of the task when the task handoff request is received. Thus, theelectronic device 102D may be prepared to receive the handoff of thetask with minimal or no latency upon receiving the task handoff request.

When the electronic device 102D is ready to receive the handoff of thetask (which may be immediately upon receiving the task handoff request),the electronic device 102D may transmit a task handoff ready indicationto the electronic device 102A to indicate that the electronic device102D is ready to receive the handoff of the task (524). Upon receivingthe task handoff ready indication (526), the electronic device 102A mayinitiate handing off the task to the electronic device 102D (528). Thehandoff of the task may include transferring task state information,such as application state information, to the electronic device 102D,such as via the server 110. In one or more implementations, the server110 may coordinate handing off the task. For example, the server 110 maymonitor/store state information corresponding to the task beingperformed at the electronic device 102A and the server 110 maycommunicate the task state information to the electronic device 102D.

The electronic device 102D receives the task state information, e.g.from the electronic device 102A and/or from the server 110, and theelectronic device 102D continues the task. When the server had alreadytransmitted the task handoff preparation request (515) to the electronicdevice 102D prior to the electronic device 102A transmitting the taskhandoff request (518), there may be little or no latency between thetime when the user requests that the task be handed off (516) and thetask continuing on the electronic device 102D (530). Thus, the user mayexperience a seamless transition from performing the task using theelectronic device 102A to continuing the task using the electronicdevice 102D.

In one or more implementations, when the electronic device 102Dinitiates the continuation of the task, the electronic device 102D maytransmit, to the electronic device 102A, a task handoff confirmationindicating that the electronic device 102D has continued the task (532).The electronic device 102A may stop performing the task when the taskhandoff confirmation is received (534). Thus, in one or moreimplementations, the task may be performed concurrently on theelectronic device 102A and the electronic device 102D for a minimalamount of time during the handoff process. The server 110 may determinethat the one of the electronic devices 102A,D that the user is currentlytouching, typing, or interacting with is the active electronic device,and the server 110 may only transmit communications to the activeelectronic device.

In one or more implementations, both of the electronic devices 102A,Dmay be concurrently considered the active electronic device. If a signalhas been determined from the activity of the user that indicates thatthe user will likely no longer be engaging with one or more of theelectronic device 102A, e.g. logging out, locking, closing, poweringdown, etc., that electronic device may no longer be considered an activeelectronic device.

FIGS. 6-14 illustrate flow diagrams of example processes 600-1400 of themulti-device communication management system in accordance with one ormore implementations. For explanatory purposes, the processes 600-1400are primarily described herein with reference to the electronic devices102A-C, and the server 110 of FIGS. 1-4, where the server 110 mayinclude and/or represent one or more of a relay server 412, an allocatorserver 414, or a notification server 416. However, the processes600-1400 are not limited to the electronic devices 102A-C and/or theserver 110, and one or more blocks (or operations) of the processes600-1400 may be performed by one or more other components of theelectronic devices 102A-C and/or the server 110. The electronic devices102A-C and/or the server 110 are also presented as exemplary devices andthe operations described herein may be performed by any suitabledevices. In one or more implementations, the electronic devices 102A-Cmay communicate with one another directly and/or via a server (e.g., theserver 110). Further for explanatory purposes, the blocks of theprocesses 600-1400 are described herein as occurring in serial, orlinearly. However, multiple blocks of the processes 600-1400 may occurin parallel. In addition, the blocks of the processes 600-1400 need notbe performed in the order shown and/or one or more of the blocks of theprocesses 600-1400 need not be performed and/or can be replaced by otheroperations.

The process 600 illustrates allocation of a session, such as anaudio/video communication session, for the electronic device 102A. Forexample, the electronic device 102A may be participating in a groupmessaging session with the electronic devices 102B-C, and the electronicdevice 102A may request that the group messaging session be transitionedto an audio/video communication session.

In the process 600, the electronic device 102A transmits an allocationrequest to the allocator server 414 (612), requesting allocation of asession, such as an audio/video communication session. The allocationrequest may include a request type indicating a multi-way allocationinvolving multiple electronic devices, a group identifier (ID) for theelectronic device 102A, a stable ID for the electronic device 102A, ashared-session indicator, and/or a destination list. The group ID is anidentifier that is assigned to the electronic device 102A when theelectronic device 102A joins a group for group communication, such as amessaging communication group, and may expire when the electronic device102A leaves the group. The stable ID is a persistent identifier for theelectronic device 102A that does not change regardless of whether theelectronic device 102A joins or leaves a group. The destination list mayidentify other devices (e.g., the electronic devices 102B and 102C,and/or other devices associated with the same user account as theelectronic device 102B or 102C) to receive an allocation response.

When the allocator server 414 receives the allocation request from theelectronic device 102A, the allocator server 414 determines whether asession (e.g., a quick relay (QR) session) exists that the electronicdevice 102A may participate in, for example, if one or more of theelectronic devices 102B-C has already initiated a session. If there isno existing session, the allocator server 414 may allocate a newsession. Thus, in response to the allocation request, the allocatorserver 414 allocates a session and transmits, to the electronic device102A, an allocation response with the credential information needed forthe electronic device 102A to join the session (614). The allocatorserver 414 may also transmit an allocation response to each of the otherelectronic devices identified on the destination list.

Thus, the allocator server 414 may transmit, to the electronic device102B, an allocation response with the credential information needed forthe electronic device 102B (e.g., Laptop) to join the session (616) andmay transmit an allocation response to another electronic device 602(e.g., a phone) that is associated with the same user account as theelectronic device 102B (618). Further, the allocator server 414 may alsotransmit, to the electronic device 102C, an allocation response with thecredential information needed for the electronic device 102C to join thesession (620). Each allocation response may include the request typeindicating the multi-way allocation, the group ID, the stable ID, and aparticipant ID for a corresponding electronic device, and a session IDidentifying the session. The credential information included in eachallocation response may include a session key and an access token for acorresponding electronic device.

The process 700 of FIG. 7 may continue after the process 600. Theprocess 700 illustrates a process for the electronic device 102A to joina session. The electronic device 102A transmits, to the relay server412, an allocation bind request with the credential information of theelectronic device 102A (712), in order to join the session using thecredential information. In response to the allocation bind request, therelay server 412 transmits an allocation bind success response to theelectronic device 102A (714), to indicate that the electronic device102A has joined the session. When the electronic device 102A receivesthe allocation bind success response, the electronic device 102Adetermines that the electronic device 102A has joined the session (716).

When the electronic device 102A has joined the session, the electronicdevice 102A transmits a join notification to members of the group viathe notification server 416. In particular, the electronic device 102Atransmits a join notification to the notification server 416 (718). Thenotification server 416 transmits the join notification of theelectronic device 102A with a session ID to the electronic device 102B(720), where the session ID identifies the session that the electronicdevice 102A has joined.

When the electronic device 102B receives the join notification, theelectronic device 102B transmits an information request to the relayserver 412 to request information about the electronic device 102A(722). The information request may include a session key and/or anaccess token, and a request for information on participants of thesession identified by the session ID included in the join notification.The information request may also include a channel number for thesession. In response, the relay server 412 transmits an informationresponse to the electronic device 102B (724), where the informationresponse may include participant IDs identifying devices that areparticipating in the session (e.g., a participant ID of the electronicdevice 102A). The information response may further include the channelnumber for the session.

Further, the notification server 416 transmits the join notification ofthe electronic device 102A with the session ID to the electronic device102C (726), where the session ID identifies the session that theelectronic device 102A has joined. When the electronic device 102Creceives the join notification, the electronic device 102C may transmitan information request to the relay server 412 to request informationabout the electronic device 102A (728). The information request mayinclude a session key and/or an access token, and a request forinformation on participants of the session identified by the session IDincluded in the join notification. The information request may alsoinclude a channel number for the session. In response, the relay server412 transmits an information response to the electronic device 102C(730), where the information response may include participant IDsidentifying devices that are participating in the session (e.g., aparticipant ID of the electronic device 102A). The information responsemay further include the channel number for the session.

The process 800 of FIG. 8 may continue after the process 700. Theprocess 800 illustrates a process for the electronic device 102B to joina session that the electronic device 102A has joined. When theelectronic device 102B attempts to join a session that was initiated bythe electronic device 102A, the electronic device 102B transmits anallocation bind request to the relay server 412 (812). The allocationbind request may include the session ID identifying the session and asession key and/or an access token used for joining the session.

The electronic device 102B may perform an allocation process for joiningthe session. The electronic device 102B transmits an allocation requestto the allocator server 414 (822) for allocation of the session. Inresponse, the allocator server 414 allocates the session and transmits,to the electronic device 102B, an allocation response with thecredential information needed for the electronic device 102A to use tojoin the session (824). The allocation response may include a session IDand a participant ID of the electronic device 102B. The allocator server414 may transmit an allocation response to the electronic device 102A(826), where the allocation response may include a participant ID of theelectronic device 102A. The allocator server 414 may also transmit anallocation response to the electronic device 102D that is associatedwith the same user account as the electronic device 102A (828), wherethe allocation response may include a participant ID of the electronicdevice 102D. Therefore, the allocation process may be performed at boththe electronic device 102A and the electronic device 102D, such that thesession can be quickly handed off from the electronic device 102A to theelectronic device 102D. Further, the allocator server 414 may transmitan allocation response to the electronic device 102B (830).

In response to the allocation bind request, the relay server 412transmits an allocation bind success response to the electronic device102B (832), to indicate that the electronic device 102B has joined thesession. When the electronic device 102B receives the allocation bindsuccess response, the electronic device 102B determines that theelectronic device 102B has joined the session (834).

In one scenario, if the access token provided via the allocation bindrequest has expired, the relay server 412 denies the allocation bindrequest and transmits an allocation bind error response to theelectronic device 102B, where the allocation bind error responseindicates that the access token has expired (842). Then, the electronicdevice 102B may transmit another allocation bind request with an updatedaccess token to the relay server 412 (844). If the relay server 412authenticates the updated access token, the relay server 412 transmitsan allocation bind success response to the electronic device 102B, toindicate that the electronic device 102B has joined the session (846).When the electronic device 102B receives the allocation bind successresponse, the electronic device 102B determines that the electronicdevice 102B has joined the session (848).

The process 900 of FIG. 9 may continue after the process 800. Theprocess 900 illustrates a process involving a join notification. Whenthe electronic device 102B has joined the session, the electronic device102B transmits a join notification to members of the group via thenotification server 416. In particular, the electronic device 102Btransmits a join notification to the notification server 416 (912). Thenotification server 416 transmits the join notification of theelectronic device 102B with the session ID to the electronic device 102A(914), where the session ID identifies the session that the electronicdevice 102B has joined.

When the electronic device 102A receives the join notification, theelectronic device 102A transmits an information request to the relayserver 412 to request information about the electronic device 102B(916). The information request may include a session key and/or anaccess token, and a request for information on participants of thesession identified by the session ID included in the join notification.The information request may also include a channel number for thesession. In response, the relay server 412 transmits an informationresponse to the electronic device 102A (918), where the informationresponse may include participant IDs identifying devices that areparticipating in the session (e.g., a participant ID of the electronicdevice 102A). The information response may further include the channelnumber for the session.

Further, the notification server 416 transmits the join notification ofthe electronic device 102B with the session ID to the electronic device102C (920), where the session ID identifies the session that theelectronic device 102B has joined. When the electronic device 102Creceives the join notification, the electronic device 102C may transmitan information request to the relay server 412 to request informationabout the electronic device 102B (922). The information request mayinclude a session key and/or an access token, and a request forinformation on participants of the session identified by the session IDincluded in the join notification. The information request may alsoinclude a channel number for the session. In response, the relay server412 transmits an information response to the electronic device 102C(924), where the information response may include participant IDsidentifying devices that are participating in the session (e.g., aparticipant ID of the electronic device 102B). The information responsemay further include the channel number for the session.

The process 1000 of FIG. 10 may continue after the process 900. Theprocess 1000 illustrates a key exchange process for the electronicdevice 102B. An electronic device in the session may share a pre-key ofthe electronic device with active participants of the session. In theprocess 1000, the active participants of the session are the electronicdevices 102A-C. The electronic device 102B transmits a pre-key of theelectronic device 102B to the notification server 416 (1012). Thenotification server 416 may transmit a pre-key response to theelectronic device 102B to indicate that the pre-key has been received(1014). When the notification server 416 receives the pre-key, thenotification server 416 forwards the pre-key to other electronic devicesin the session. In particular, the notification server 416 forwards thepre-key to the electronic device 102A (1016) and to the electronicdevice 102C (1018). The notification server 416 may transmit a pre-keyresponse to the electronic device 102B to indicate that the pre-key hasbeen received (1014).

After transmitting the pre-key to the other electronic devices via thenotification server 416, the electronic device 102B generates a masterkey based on the pre-key and transmits the master key to thenotification server 416 (1020). The notification server 416 may transmita master key response to the electronic device 102B to indicate that themaster key has been received (1022).

After the master key has been provided to the notification server 416,the electronic device 102B may encrypt data with the master key (1024).The electronic device 102B may transmit the encrypted data to anotherdevice via the relay server 412. For example, the electronic device 102Bmay transmit the encrypted data to the relay server 412 (1026), and therelay server 412 may forward the encrypted data to the electronic device102A (1028). The electronic device 102A may not yet have the master keyto decrypt the encrypted data (1030). The notification server 416transmits the master key to the electronic device 102A (1032). Then, theelectronic device 102A may decrypt the encrypted data using the masterkey (1034).

When both of the electronic devices 102A and 102B have the master key,the electronic devices 102A and 102B may perform encrypted datacommunication with each other. For example, the electronic device 102Bmay transmit another encrypted data to the relay server 412 (1036), andthe relay server 412 may forward the encrypted data to the electronicdevice 102A (1038). The electronic device 102A may transmit anotherencrypted data to the relay server 412 (1040), and the relay server 412may forward the encrypted data to the electronic device 102B (1042).

The process 1100 of FIG. 11 may continue after the process 1000. Theprocess 1100 illustrates an allocation bind process for the electronicdevice 102C. The electronic device 102C may perform an allocationprocess for joining the session started by the electronic device 102A.In particular, the electronic device 102C transmits an allocationrequest to the allocator server 414 (1112). The allocator server 414 maytransmit an allocation response to indicate that the allocator server414 has received the allocation request (1114). In response to theallocation request, the allocator server 414 transmits, to theelectronic device 102C, an allocation response with the credentialinformation needed for the electronic device 102C to use to join thesession (1116). The allocator server 414 may transmit, to the electronicdevice 102A, an allocation response with the credential informationneeded for the electronic device 102C to use to join the session (1118).Further, the allocator server 414 may transmit, to the electronic device102B, an allocation response with the credential information needed forthe electronic device 102C to use to join the session (1120).

The electronic device 102C may transmit, to the relay server 412, anallocation bind request with the credential information of theelectronic device 102C (1122), in order to join the session using thecredential information (1122). In response to the allocation bindrequest, the relay server 412 transmits an allocation bind successresponse to the electronic device 102C (1124), to indicate that theelectronic device 102C has joined the session. When the electronicdevice 102C receives the allocation bind success response, theelectronic device 102C determines that the electronic device 102C hasjoined the session (1126).

In one scenario, if the access token provided via the allocation bindrequest has expired, the relay server 412 denies the allocation bindrequest and transmits an allocation bind error response to theelectronic device 102C, where the allocation bind error responseindicates that the access token has expired (1128). Then, the electronicdevice 102C may transmit another allocation bind request with an updatedaccess token to the relay server 412 (1130). If the relay server 412authenticates the updated access token, the relay server 412 transmitsan allocation bind success response to the electronic device 102C, toindicate that the electronic device 102C has joined the session (1132).When the electronic device 102C receives the allocation bind successresponse, the electronic device 102C determines that the electronicdevice 102C has joined the session (1134).

The process 1200 of FIG. 12 may continue after the process 1100. Theprocess 1200 illustrates a join process for the electronic device 102Cto become an active participant of a group communication started by oneor more of the electronic devices 102A and 102B. The electronic device102C transmits a join message to the notification server 416 (1212). Inresponse, the notification server 416 may transmit a join response tothe electronic device 102C to indicate that the notification server 416has received the join message (1214).

The notification server 416 forwards the join message to the otherelectronic devices 102A-D. In particular, the notification server 416forwards the join message to the electronic device 102A (1216). Inresponse, the electronic device 102A updates its active participant listto include the electronic devices 102B and 102C as active participantswith the electronic device 102A (1218). The notification server 416forwards the join message to the electronic device 102B (1220). Inresponse, the electronic device 102B updates its active participant listto include the electronic devices 102A and 102C as active participantswith the electronic device 102B (1222).

Further, to update the active participant list of the electronic device102C, the other devices may send active participant notifications to theelectronic device 102C. In particular, the electronic device 102Atransmits an active participant notification to the notification server416 (1224). The notification server 416 may transmit an activeparticipant notification response to the electronic device 102A toindicate that the notification server 416 has received the activeparticipant notification (1226). The notification server 416 forwardsthe active participant notification to the electronic device 102C(1228). Then, the electronic device 102C updates its active participantlist to indicate that the electronic device 102A is an activeparticipant with the electronic device 102C (1230).

Subsequently, the electronic device 102B transmits an active participantnotification to the notification server 416 (1232). The notificationserver 416 may transmit an active participant notification response tothe electronic device 102B to indicate that the notification server 416has received the active participant notification (1234). Thenotification server 416 forwards the active participant notification tothe electronic device 102C (1236). Then, the electronic device 102Cupdates its active participant list to indicate that the electronicdevices 102A and 102B are active participants with the electronic device102C (1238).

The process 1300 of FIG. 13 may continue after the process 1200. Theprocess 1300 illustrates a key exchange process for the electronicdevice 102C. An electronic device in the session may share a pre-key ofthe electronic device with the active participants. In the process 1200,the active participants are the electronic devices 102A-C, and thus theelectronic device 102C may share a pre-key of the electronic device 102Cwith the electronic devices 102A-B. In particular, the electronic device102C transmits the pre-key of the electronic device 102C to thenotification server 416 (1312). In response, the notification server 416may transmit a pre-key response to the electronic device 102C toindicate that the notification server 416 has received the pre-key(1314). The notification server 416 forwards the pre-key of theelectronic device 102C to the electronic device 102A (1316) and to theelectronic device 102B (1318).

The electronic device 102C generates a master key based on a pre-key ofthe electronic device 102A and sends the master key to the notificationserver 416 (1320). In response, the notification server 416 may transmita master key response to the electronic device 102C to indicate that thenotification server 416 has received the master key (1322). Theelectronic device 102C generates a master key based on a pre-key of theelectronic device 102B and sends the master key to the notificationserver 416 (1324). In response, the notification server 416 may transmita master key response to the electronic device 102C to indicate that thenotification server 416 has received the master key (1326).

The electronic device 102C may encrypt data using the master key of theelectronic device 102C (1328). The electronic device 102C may transmitthe encrypted data to the relay server 412 (1330), and the relay server412 then forwards the encrypted data to the electronic device 102A(1332) and to the electronic device 102B (1334). The notification server416 transmits, to the electronic device 102A, the master key of theelectronic device 102C based on the pre-key of the electronic device102A (1336). The electronic device 102A decrypts the encrypted datausing the master key (1338). The notification server 416 transmits, tothe electronic device 102B, the master key of the electronic device 102Cbased on the pre-key of the electronic device 102B (1340). Theelectronic device 102B decrypts the encrypted data using the master key(1342).

After distributing the master key, the electronic devices 102A-C mayperform encrypted communication with one another. For example, theelectronic device 102B transmits an encrypted data to the relay server412 (1344), and the relay server 412 then forwards the encrypted data tothe electronic device 102A (1346) and to the electronic device 102C(1348). For example, the electronic device 102A transmits an encrypteddata to the relay server 412 (1350), and the relay server 412 thenforwards the encrypted data to the electronic device 102B (1352) and tothe electronic device 102C (1354).

The process 1400 of FIG. 14 may continue after the process 1300. Theprocess 1400 illustrates a process when the electronic device 102C isrequested to leave the group communication. The electronic device 102Amay request the electronic device 102C to leave the group communication,which prompts the electronic device 102C to transmit an unbindallocation request to the relay server 412 (1412). In response, therelay server 412 transmits an unbind allocation response to theelectronic device 102C (1414) to indicate that the electronic device102C is no longer in the session for the group communication. Then, theelectronic device 102C disconnects the link for the group communicationwith the electronic devices 102A and 102B (1416)

After an electronic device leaves the session, other electronic devicesin the session may update, or roll, the master key, and/or any othershared keys, such that the electronic device that is no longer in thesession will not be able to join the session or communicate with theother devices. In the process 1400, the notification server 416transmits, to the notification server 416, an updated master key of theelectronic device 102A based on the pre-key of the electronic device102B (1418). The notification server 416 may transmit a master keyresponse to the electronic device 102A to indicate that the updatedmaster key has been received (1420). The electronic device 102A mayencrypt data with the updated master key (1422). The notification server416 forwards the updated master key to the electronic device 102B(1424). The electronic device 102B may encrypt data with the updatedmaster key (1426).

Thus, the electronic devices 102A and 102B may use the updated masterkey to encrypt data for communication with each other. For example, theelectronic device 102A transmits an encrypted data to the relay server412 (1428), and the relay server 412 then forwards the encrypted data tothe electronic device 102B (1430). For example, the electronic device102B transmits an encrypted data to the relay server 412 (1432), and therelay server 412 then forwards the encrypted data to the electronicdevice 102A (1434).

The electronic device 102C may notify the group that the electronicdevice 102C has left. In particular, the electronic device 102Ctransmits a leave notification to the notification server 416 (1436).The notification server 416 may transmit a leave notification responseto the electronic device 102C (1438) to indicate that the notificationserver 416 has received the leave notification. Then, the notificationserver 416 forwards the leave notification to the electronic device 102A(1440). In response to the leave notification, the electronic device102A updates the active participant list to remove the electronic device102C from the active participant list (1442). The notification server416 also forwards the leave notification to the electronic device 102B(1444). In response to the leave notification, the electronic device102A updates the active participant list to remove the electronic device102C from the active participant list (1446).

In one or more implementations, one or more of the messages and/orrequests described above in FIGS. 6-14 may be, and/or may include, anarray. For example, identity requests/queries may be transmitted as anarray, such that multiple identity requests can be packaged/transmittedin a consolidated format via a single array, thereby reducing the amountof time and/or overhead required for session setup/establishment. Thus,to send identity requests corresponding to four differententities/devices, the electronic device 102A only needs to send onearray containing four different identity requests, instead of sendingfour separate identity requests. For example, the destination list ofthe allocation request may be an array that stores identifiers of eachof the other electronic devices 102B-C, thereby allowing the electronicdevice 102A to transmit a single allocation request to initialize thesession with multiple other electronic devices 102B-C.

FIG. 15 illustrates a flow diagram of an example process 1500 of themulti-device communication management system in accordance with one ormore implementations. For explanatory purposes, the process 1500 isprimarily described herein with reference to the electronic devices102A-C, and the server 110 of FIGS. 1-3. The electronic devices 102A-Cand the server 110 are also presented as exemplary devices and theoperations described herein may be performed by any suitable devices.The electronic devices 102A-C may communicate with one another directlyand/or via a server (e.g., the server 110). Further for explanatorypurposes, the blocks of the process 1500 are described herein asoccurring in serial, or linearly. However, multiple blocks of theprocess 1500 may occur in parallel. In addition, the blocks of theprocess 1500 need not be performed in the order shown and/or one or moreof the blocks of the process 1500 need not be performed and/or can bereplaced by other operations.

In the process 1500, the electronic devices 102A-C may be participatingin, or preparing to participate in, and audio/video group communicationsession. One or more of the electronic devices 102A-C and/or the server110, such as the electronic device 102A, receives buffer level report(s)from the other electronic devices 102B-C in the group communicationsession (1512, 1514). A buffer level report may include, for example,information regarding one or more buffers, such as one or more jitterbuffers at a given electronic device 102B. For example, the electronicdevice 102B may maintain a jitter buffer for audio and/or video packetsreceived from the electronic device 102A, as well as a separate jitterbuffer for audio and/or video packets received from the electronicdevice 102C. The jitter buffer may be used to avoid buffer underruns dueto jitter and/or propagation delays. The information regarding the oneor more buffers may include, for example, the current target bufferlevel being maintained, a range of the high and low buffer levels for agiven period of time, and the like.

In one or more implementations, the current target buffer level beingmaintained by a given electronic device 102B for audio and/or videopackets received from another electronic device 102A may be indicativeof the audio and/or video output delay experienced by the user of theelectronic device 102B with respect to the audio and/or video streamcorresponding to the user of the electronic device 102A. For example,the higher the current target buffer level is, the greater the audioand/or video output delay may be, since a larger number of audio and/orvideo packets are being buffered. In one or more implementations, theelectronic devices 102B-C may provide the buffer level reports to theelectronic device 102A on a continuous and/or periodic basis.

The electronic device 102A may process the buffer level reports todetermine a group target buffer level, and/or a group target bufferlevel range, to be implemented by each of the electronic devices 102A-Cfor each of the jitter buffers (1518). In this manner, the target bufferlevel, and corresponding audio and/or video output delay, will besubstantially the same for each of the electronic devices 102A-C,thereby ensuring fairness in the communications of the audio/video groupcommunication session, e.g., since each of the users will hear/see agiven audio/video stream at approximately the same time and thereforehave an equal opportunity to respond.

The electronic device 102A may determine the group target buffer level,for example, such that the group target buffer level is within, orabove, the range of high and low buffer levels indicated by each of theelectronic devices 102B-C, as well as the range of high and low bufferlevels of the electronic device 102A. In one or more implementations,the electronic device 102A may determine the group target buffer levelwith an upper bound such that the audio and/or video delay experiencedby the users of the electronic devices 102A-C is not excessively long.

The electronic device 102A may transmit an indication of the grouptarget buffer level to the electronic device 102B (1520) and theelectronic device 102C (1522). The electronic device 102A maysubsequently receive one or more downlink communications from theelectronic device 102B (1524), and one or more downlink communicationsfrom the electronic device 102C (1526), and the electronic device 102Amay implement the group target buffer level (1528) for the receivedcommunications.

Similarly, the electronic device 102B may receive one or more downlinkcommunications from the electronic device 102A (1530) and the electronicdevice 102C (1532), and the electronic device 102B may implement thegroup target buffer level (1534) for the received communications. Theelectronic device 102C may receive one or more downlink communicationsfrom the electronic device 102A (1536) and the electronic device 102B(1538), and the electronic device 102C may implement the group targetbuffer level (1540) for the received communications.

In one or more implementations, the server 110 may receive the bufferlevel reports from the electronic devices 102A-C, and the server 110 maydetermine the appropriate group target buffer level for the electronicdevices 102A-C. In one or more implementations, the group target bufferlevel may be indicated as a percentage of the buffer that is occupiedwith data, a number of bytes in the buffer, etc.

In one or more implementations, the group target buffer level may beimplemented as a function of the audio delay experienced at each of theelectronic devices 102A-C. For example, the electronic devices 102A-Cmay share their audio delays with each other, and the electronic devices102A-C may agree on an group audio delay. For example, the electronicdevices 102A-C may agree to implement an audio delay that is equal tothe greatest audio delay being experienced by one of the electronicdevices 102A-C, such as the electronic device 102A. Thus, the electronicdevice 102A may not implement any additional audio delay, while theelectronic devices 102B-C may implement an additional audio delay sothat their total audio delay equals the audio delay experienced by theelectronic device 102A.

FIG. 16 illustrates a flow diagram of an example process 1600 of themulti-device communication management system in accordance with one ormore implementations. For explanatory purposes, the process 1600 isprimarily described herein with reference to the electronic devices102A-C and the server 110 of FIGS. 1-3. However, the electronic devices102A-C and the server 110 are presented as exemplary devices and theoperations described herein may be performed by any suitable devices.The electronic devices 102A-C may communicate with one another directlyand/or via a server (e.g., the server 110). Further for explanatorypurposes, the blocks of the process 1600 are described herein asoccurring in serial, or linearly. However, multiple blocks of theprocess 1600 may occur in parallel. In addition, the blocks of theprocess 1600 need not be performed in the order shown and/or one or moreof the blocks of the process 1600 need not be performed and/or can bereplaced by other operations.

In the process 1600, the electronic devices 102A-C may be participatingin, or preparing to participate in, and audio/video group communicationsession, such as in coordination with the server 110. The server 110 mayreceive downlink condition reports from electronic devices 102A-C (1612,1614, 1616) and determine a quality of the audio and/or video streams tobe transmitted by each of the electronic devices 102A-C to each of theother electronic devices 102A-C for the audio/video group communicationsession (1622).

The downlink condition reports may indicate a downlink condition fromthe server 110 to the respective electronic devices 102A-C, such asdownlink bandwidth, downlink signal to noise ratio (SNR), downlink biterror rate (BER), downlink received signal strength indication (RSSI),and the like. In one or more implementations, the downlink conditionreports may be, and/or may include Real-time Transport Protocol (RTP)Control Protocol (RTCP) reports. The server 110 may also generate uplinkcondition reports for the uplink channels between the electronic devices102A-C and the server 110. In one or more implementations, the uplinkand downlink channels may be asymmetrical. For example, the electronicdevice 102A may have a downlink via Wi-Fi and have an uplink viacellular. Thus, the uplink and downlink channel conditions/bandwidthsmay differ for a given electronic device 102A.

Based on the downlink channel reports received by the server 110, andthe uplink channel reports generated by the server 110, the server maydetermine appropriate tiers of audio and/or video streams to betransmitted and/or made available by each of the electronic devices102A-C. For example, the server 110 may determine a number of tiers ofquality levels of audio and/or video streams, such as three tiers, foreach of the electronic devices 102A-C to provide and/or make availableto the other electronic devices 102A-C.

The tiers for each of the electronic devices 102A-C may be determined bythe server 110 such that high quality audio and/or video streams aremade available to the electronic devices 102A-C that have the downlinkbandwidth to support the high quality audio and/or video streams whilealso making low quality audio and/or video streams available to theelectronic devices 102A-C that have downlink bandwidth constraints. Inaddition, the server 110 may account for the uplink channel conditionsfrom a given electronic device 102A to the server 110 when determiningthe quality tiers for the given electronic device 102A. For example, theserver 110 may determine the different quality tiers for the electronicdevice 102A such that the electronic device 102A has sufficient uplinkbandwidth to support concurrently transmitting the streams at all threequality levels.

The server 110 may then transmit indications of the determined qualitytiers to the electronic devices 102A-C, respectively (1624, 1626, 1628).The electronic devices 102A-C may then transcode and/or encode theiraudio and/or video streams to reflect each of the indicated qualitytiers. For example, the electronic device 102A may encode a first videostream for a first tier at 1080P resolution, a second video stream for asecond tier at 720P resolution, and a third video stream for a thirdtier at 480i resolution. Thus, the bit rates of the tiers may differ toaccount for the varying downlink conditions for each of the otherelectronic devices 102B-C. The electronic device 102A may then transmitall of the tiers of the audio and/or video streams to the server 110 andthe server 110 may relay the appropriate streams to the other electronicdevices 102B-C, such as based on the downlink channel conditions of eachof the other electronic devices 102B-C. In one or more implementations,the server 110 may transmit an indication to the electronic device 102Aindicating which of the audio and/or video streams will be forwarded tothe other electronic devices 102B-C, and the electronic device 102A mayonly transmit the indicated audio and/or video streams to the server110, and/or directly to the other electronic devices 102B-C.

In one or more implementations, the electronic devices 102A-C mayexchange downlink and uplink channel reports to each other independentof the server 110. The electronic devices 102A-C may then eachdetermine, e.g. independent of the server 110, the appropriate tiers ofquality levels to make available to the other electronic devices 102A-C.In one or more implementations, the electronic devices 102A-C mayadvertise to the different quality tiers they have available to eachother. For example, the electronic device 102A may broadcast, and/orpublish to the server 110, a manifest file to the electronic devices102B-C that lists the audio and/or video profiles of the audio and/orvideo streams that the electronic device 102A can provide. The profilesmay include information such as bit rate, codec, resolution, frame rate,and the like. The other electronic devices 102B-C may receive themanifest and may transmit a message to the electronic device 102Arequesting one of the available audio and/or video streams. Theelectronic device 102A may transmit the requested audio and/or videostreams to the electronic devices 102B-C, such as directly or throughthe server 110.

In one or more implementations, the server 110 and/or the electronicdevices 102A-C may continuously monitor the uplink and/or downlinkchannel reports and may dynamically and/or adaptively change the tiersof the quality levels being made available by one or more of theelectronic devices 102A-C. Thus, as the network conditions change, ormore electronic devices join the group communication session, the tiersof the quality levels are dynamically and/or adaptively adjusted.

In one or more implementations, the server 110 and/or the electronicdevices 102A-D may cap the number of audio and/or video streams beingreceived by a given electronic device 102A at any given time. Forexample, if the electronic device 102A has very little downlinkbandwidth available, the server 110 may only transmit a single audioand/or video stream to the electronic device 102A, such as an audioand/or video stream corresponding to the user who is currently speakingor communicating. The electronic device 102A may display static imagesfor the other users in the audio/video group communication session.

In one or more implementations, if the electronic device 102A only hassufficient bandwidth to receive an audio stream, e.g. without a videostream, the electronic device 102A may animate an images (or avatars) ofthe users associated with the other electronic devices 102B-C tocoincide with when the users associated with the other electronicdevices 102B-C are speaking. Thus, the mouths of the users in the images(and/or the mouths of the avatars) may open and close in synchronicitywith the words being spoken by the other users.

In one or more implementations, the electronic device 102A may move froma first network (e.g., WLAN) that provides high uplink transmission rateto a second network (e.g., cellular network) that provides lower uplinktransmission rate. In such an example, the electronic device 102A may beconfigured to transmit high quality audio and/or video streams viauplink communication while in the first network, and to transmit data oflow quality audio and/or video streams via uplink communication while inthe second network. In one example, even if the electronic device 102Ais capable of transmitting a higher quality audio and/or video streamswhile in the second network, the electronic device 102A may beconfigured to transmit the low quality audio and/or video streamsregardless of the uplink condition, in case the uplink conditiondeteriorates to a point where the higher quality audio and/or videosstreams cannot be reliably transmitted in the uplink communicationand/or in case downlink conditions/bandwidth to the other electronicdevices 102B-C are not sufficient for communication of the higherquality audio and/or videos streams.

FIGS. 17 and 18 illustrate flow diagrams of example processes 1700 and1800 of the multi-device communication management system in accordancewith one or more implementations. For explanatory purposes, theprocesses 1700 and 1800 are primarily described herein with reference tothe electronic device 102A of FIGS. 1-2. However, the processes 1700 and1800 are not limited to the electronic device 102A, and one or moreblocks (or operations) of the processes 1700 and 1800 may be performedby one or more other components of the electronic device 102A. Theelectronic device 102A also is presented as an exemplary device and theoperations described herein may be performed by any suitable device,such as one or more of the other electronic devices 102B-D. Further forexplanatory purposes, the blocks of the processes 1700 and 1800 aredescribed herein as occurring in serial, or linearly. However, multipleblocks of the processes 1700 and 1800 may occur in parallel. Inaddition, the blocks of the processes 1700 and 1800 need not beperformed in the order shown and/or one or more of the blocks of theprocesses 1700 and 1800 need not be performed and/or can be replaced byother operations.

The process 1700 describes a handoff of a task from the perspective ofthe electronic device 102A that is handing off the task. In the process1700, the electronic device 102A transmits, to the server 110,information corresponding to at least one of a task in progress, devicecapability, or user attention (1702). For example, the electronic device102A may transmit information regarding a task in progress, e.g. a taskfor which a handoff request may subsequently be received from the user,information regarding the capabilities of the electronic device 102A,and/or user attention information that is obtainable and/or determinableat the electronic device 102A.

The electronic device 102A may receive, from the server 110, anindication of the electronic devices that are registered to the sameuser account as the electronic device 102A and are available to receivea handoff of the task (1703). The electronic device 102A may provide theuser with a list of the other electronic devices that are available tohave a task handed off to.

The electronic device 102A may receive a request to handoff a task to asecond electronic device, such as the electronic device 102D (1704). Forexample, the electronic device 102A may receive a request from the userto handoff the task to the electronic device 102D. Upon receiving therequest to handoff the task (1704), the electronic device 102Atransmits, to the second electronic device, such as the electronicdevice 102D, a task handoff request requesting that the secondelectronic device to prepare to receive a handoff of the task (1706).

Once the second electronic device is ready to receive the task, theelectronic device 102A receives a task handoff ready indication from thesecond electronic device, such as the electronic device 102D (1710).When the second electronic device is prepared to receive the handoff ofthe task, the electronic device 102A initiates the handoff of the taskto the second device (1714).

The electronic device 102A determines whether the second electronicdevice has initiated the continuation of the task (1716). In one or moreimplementations, the electronic device 102A determines whether thesecond electronic device has initiated the continuation of the task bydetermining whether a task continuation confirmation has been receivedfrom the second electronic device, the task continuation confirmationindicating that the second electronic device has initiated thecontinuation of the task. The electronic device 102A may be configuredto determine that the second electronic device has initiated thecontinuation of the task when the task continuation confirmation isreceived. If the second electronic device has initiated the continuationof the task (1716), the electronic device 102A may stop performing thetask (1718). If the second electronic device has not initiated thecontinuation of the task (1716), the electronic device 102A continues toperform the task (1720) at least until determining that the secondelectronic device has initiated the continuation of the task.

The process 1800 describes a handoff of a task from the perspective ofthe electronic device 102D that is receiving the handoff of the task. Inthe process 1800, the electronic device 102D may transmit, to the server110 (and/or the electronic device 102A), information corresponding to atleast one of a capability of the electronic device 102D or userattention information (1802). The electronic device 102D receives, fromthe server 110 (and/or the electronic device 102A), a task handoffpreparation request (1804). The task handoff preparation request may bereceived by the electronic device 102D in anticipation of a task beinghanded off to the electronic device 102D but before a handoff of thetask has been requested. In response receiving the task handoffpreparation request, the electronic device 102D performs preliminarypreparations for receiving a possible handoff of the task from theelectronic device 102A (1806).

The electronic device 102D determines whether a task handoff request hasbeen received, such as from the electronic device 102A and/or the server110 (1808). When the task handoff request is received (1808), theelectronic device completes any remaining preparations for receiving ahandoff of the task. For example, the task handoff request may includeinformation, such as task state information and/or application stateinformation that was not available when the task handoff preparationrequest was received. Thus, in one or more implementations, theelectronic device 102D may complete any remaining preparations forreceiving the handoff of the task based at least in part on informationreceived in the task handoff request.

When the electronic device 102D has completed the preparations toreceive the handoff of the task, the electronic device 102D maytransmit, to the electronic device 102A and/or the server 110, a taskhandoff ready indication indicating that the electronic device 102D isready to receive the handoff of the task (1810). The electronic device102D receives a handoff of the task from the electronic device 102Aand/or the server 110 (1812). The handoff of the task may include taskand/or application state information that may be used by the electronicdevice 102D to initiate the continuation of the task (1814). Afterinitiating the continuation of the task, the electronic device 102D maytransmit, to the server 110 and/or the electronic device 102A, a taskcontinuation confirmation (1816).

FIGS. 19-23 illustrate flow diagrams of example processes 1900-2300 ofthe multi-device communication management system in accordance with oneor more implementations. For explanatory purposes, the processes1900-2300 are primarily described herein with reference to theelectronic device 102A of FIGS. 1-3. However, the processes 1900-2300are not limited to the electronic device 102A, and one or more blocks(or operations) of the processes 1900-2300 may be performed by one ormore other components of the electronic device 102A. The electronicdevice 102A also is presented as an exemplary device and the operationsdescribed herein may be performed by any suitable device, such as one ormore of the other electronic devices 102B-D and/or the server 110.Further for explanatory purposes, the blocks of the processes 1900-2300are described herein as occurring in serial, or linearly. However,multiple blocks of the processes 1900-2300 may occur in parallel. Inaddition, the blocks of the processes 1900-2300 need not be performed inthe order shown and/or one or more of the blocks of the processes1900-2300 need not be performed and/or can be replaced by otheroperations.

In the process 1900, the electronic device 102A transmits, to anallocator server 414, an allocation request requesting allocation of asession (1902). The allocation request may include a group ID of theelectronic device 102A for the session, a stable ID of the electronicdevice 102A. The allocation request may further include a destinationdevice list identifying the one or more second devices to receive anallocation response. The allocation request may further include asession ID of the session and a participant ID of the electronic device102A.

The electronic device 102A receives, from the allocator server 414, anallocation response in response to the allocation request, theallocation response including credential information for the electronicdevice 102A to use to join the session (1904). The credentialinformation may include at least one of a session key or an access tokenfor the electronic device 102A.

The electronic device 102A transmits, to a relay server 412, anallocation bind request with the credential information to join thesession using the credential information (1906). The allocation bindrequest may include a session ID of the session and the credentialinformation. The electronic device 102A receives, from the relay server412, an allocation bind success response in response to the allocationbind request, the allocation bind success response indicating that theelectronic device 102A has joined the session (1908). The electronicdevice 102A transmits a join notification to one or more otherelectronic devices 102B-C via a notification server 416 to notify theone or more other electronic devices 102B-C that the electronic device102A has joined the session (1910).

FIG. 20 illustrates a flow diagram of an example process 2000 of themulti-device communication management system in accordance with one ormore implementations, continuing from FIG. 19. At 1912, the electronicdevice 102A continues from 1912 of FIG. 19. The electronic device 102Areceives a second join notification from at least one of the otherelectronic devices 102B-C (2002), where the second join notificationnotifies the electronic device 102A that the at least one of the otherelectronic devices 102B-C has joined the session. The electronic device102A transmits, to the relay server 412, an information requestrequesting information on the at least one of the other electronicdevices 102B-C (2004). The electronic device 102A receives, from therelay server 412, an information response including the information onthe at least one of the other electronic devices 102B-C in response tothe information request (2006).

FIG. 21 illustrates a flow diagram of an example process 2100 of themulti-device communication management system in accordance with one ormore implementations, continuing from FIG. 19. At 1912, the electronicdevice 102A continues from 1912 of FIG. 19. The electronic device 102Areceives a pre-key from at least one of the one or more other electronicdevices 102B-C (2102). The electronic device 102A receives a master keyfrom the at least one of the one or more other electronic devices102B-C, the master key being based on the pre-key (2104).

The electronic device 102A communicates data with the at least one ofthe one or more other electronic devices 102B-C using the master key,where the data is encrypted with the master key (2106). For example, theelectronic device 102A may encrypt data with the master key and transmitthe encrypted data to the at least one of the one or more otherelectronic devices 102B-C. For example, the electronic device 102A mayreceive encrypted data from the at least one of the one or more otherelectronic devices 102B-C, and decrypt the encrypted data using themaster key.

FIG. 22 illustrates a flow diagram of an example process 2200 of themulti-device communication management system in accordance with one ormore implementations, continuing from FIG. 19. At 1912, the electronicdevice 102A continues from 1912 of FIG. 19. The electronic device 102Atransmits a pre-key to at least one of the one or more other electronicdevices 102B-C that has joined the session (2202). The electronic device102A generates a master key based on the pre-key (2204). The electronicdevice 102A transmits the master key to the at least one of the one ormore other electronic devices 102B-C (2206).

The electronic device 102A communicates data with the at least one ofthe one or more other electronic devices 102B-C using the master key,where the data is encrypted with the master key (2208). For example, theelectronic device 102A may encrypt data with the master key and transmitthe encrypted data to the at least one of the one or more otherelectronic devices 102B-C. For example, the electronic device 102A mayreceive encrypted data from the at least one of the one or more otherelectronic devices 102B-C, and may decrypt the encrypted data using themaster key.

FIG. 23 illustrates a flow diagram of an example process 2300 of themulti-device communication management system in accordance with one ormore implementations, continuing from FIG. 19. At 1912, the electronicdevice 102A continues from 1912 of FIG. 19. The electronic device 102Atransmits, to the relay server 412, an allocation unbind request toleave the session (2302). The electronic device 102A receives, from therelay server 412, an allocation unbind response in response to theallocation unbind request to indicate that the electronic device 102Ahas left the session (2304). The electronic device 102A transmits aleave notification to the one or more other electronic devices 102B-Cthat have joined the session via the notification server 416 (2306). Theleave notification notifies the one or more other electronic devices102B-C that the electronic device 102A has left the session.

FIGS. 24-27 illustrate flow diagrams of example processes 2400-2700 ofthe multi-device communication management system in accordance with oneor more implementations. For explanatory purposes, the processes2400-2700 are primarily described herein with reference to the server110 of FIGS. 1 and 3-4. However, the processes 2400-2700 are not limitedto the server 110, and one or more blocks (or operations) of theprocesses 2400-2700 may be performed by one or more other components ofthe server 110. The server 110 also is presented as an exemplary deviceand the operations described herein may be performed by any suitabledevice. Further for explanatory purposes, the blocks of the processes2400-2700 are described herein as occurring in serial, or linearly.However, multiple blocks of the processes 2400-2700 may occur inparallel. In addition, the blocks of the processes 2400-2700 need not beperformed in the order shown and/or one or more of the blocks of theprocesses 2400-2700 need not be performed and/or can be replaced byother operations.

In the process 2400, the server 110 receives, from a first electronicdevice 102A, an allocation request requesting allocation of session(2402). The allocation request may include a group ID of the firstelectronic device 102A for the session, a stable ID of the first device.The allocation request may further include a destination device listidentifying one or more second electronic devices 102B-C to receive anallocation response. The allocation response may further include asession ID of the session and a participant ID of the first electronicdevice 102A.

The server 110 transmits an allocation response to the first electronicdevice 102A and one or more second electronic devices 102B-C in responseto the allocation request, the allocation response including credentialinformation for the first electronic device 102A to use to join thesession (2404). The credential information may include at least one of asession key or an access token for the first electronic device 102A.

The server 110 receives, from the first electronic device 102A, anallocation bind request with the credential information to join thesession using the credential information (2406). The allocation bindrequest may include a session ID of the session and the credentialinformation. The server 110 transmits, to the first electronic device102A, an allocation bind success response in response to the allocationbind request, the allocation bind success response indicating that thefirst electronic device 102A has joined the session (2408).

FIG. 25 illustrates a flow diagram of an example process 2500 of themulti-device communication management system in accordance with one ormore implementations, continuing from FIG. 24. At 2410, the server 110continues from 2410 of FIG. 24. The server 110 receives, from the firstelectronic device 102A, a join notification notifying that the firstelectronic device 102A has joined the session (2502). The server 110forwards the join notification to the one or more second electronicdevices 102B-C (2504). The server 110 receives, from at least one of theone or more second electronic devices 102B-Cs, an information requestrequesting information on the first electronic device 102A (2506). Theserver 110 transmits, to the at least one of the one or more secondelectronic devices 102B-C, an information response including theinformation on the first electronic device 102A in response to theinformation request (2508).

FIG. 26 illustrates a flow diagram of an example process 2600 of themulti-device communication management system in accordance with one ormore implementations, continuing from FIG. 24. At 2410, the server 110continues from 2410 of FIG. 24. The server 110 receives a pre-key fromat least one of the one or more second electronic devices 102B-C (2602).The server 110 forwards the pre-key to the first electronic device 102A(2604). The server 110 receives a master key from the at least one ofthe second electronic devices 102B-C, the master key being based on thepre-key (2606). The server 110 forwards the master key to the firstelectronic device 102A (2608). The server 110 manage communicationbetween the first electronic device 102A and the at least one of the oneor more second electronic devices 102B-C based on the master key, wherethe communication is encrypted with the master key (2610).

FIG. 27 illustrates a flow diagram of an example process 2700 of themulti-device communication management system in accordance with one ormore implementations, continuing from FIG. 24. At 2410, the server 110continues from 2410 of FIG. 24. The server 110 receives, from the firstelectronic device 102A, an allocation unbind request to leave thesession (2702). The server 110 transmits, to the first electronic device102A, an allocation unbind response in response to the allocation unbindrequest to indicate that the first electronic device 102A has left thesession (2704). The server 110 transmits a leave notification to the oneor more second electronic devices 102B-C that have joined the session,the leave notification notifying that the electronic device 102A hasleft the session (2706).

FIG. 28 illustrates a flow diagram of an example process 2800 of themulti-device communication management system in accordance with one ormore implementations. For explanatory purposes, the process 2800 isprimarily described herein with reference to the electronic device 102Aof FIGS. 1-3. However, the process 2800 is not limited to the electronicdevice 102A, and one or more blocks (or operations) of the process 2800may be performed by one or more other components of the electronicdevice 102A. The electronic device 102A also is presented as anexemplary device and the operations described herein may be performed byany suitable device, such as one or more of the other electronic devices102B-D, and/or the server 110. Further for explanatory purposes, theblocks of the process 2800 are described herein as occurring in serial,or linearly. However, multiple blocks of the process 2800 may occur inparallel. In addition, the blocks of the process 2800 need not beperformed in the order shown and/or one or more of the blocks of theprocess 2800 need not be performed and/or can be replaced by otheroperations.

The electronic device 102A receives buffer level reports from the otherelectronic devices 102B-C, respectively, that are participating in anaudio and/or video group communication session (2802). In one or moreimplementations, the buffer level reports may indicate the currenttarget buffer levels of each of the electronic devices 102A-C. Theelectronic device 102A determines a group target buffer level toequalize audio delays across the electronic devices 102A-C based atleast in part on the received buffer level reports and the currenttarget buffer level of the electronic device 102A (2806).

The electronic device 102A transmits an indication of the group targetbuffer level to each of the other electronic devices 102B-C (2808). Theelectronic device 102A receives downlink communications from the otherelectronic devices 102B-C (2810), and the electronic device 102A appliesthe group target buffer level to the buffers, such as jitter buffers,corresponding to the downlink communications received from the otherelectronic devices 102B-C (2812). In one or more implementations, theelectronic device 102A may perform the process 2800 repeatedly, orcontinuously, throughout the audio and/or video group communicationsession.

FIG. 29 illustrates a flow diagram of an example process 2900 of themulti-device communication management system in accordance with one ormore implementations. For explanatory purposes, the process 2900 isprimarily described herein with reference to the server 110 of FIGS.1-3. However, the process 2900 is not limited to the server 110, and oneor more blocks (or operations) of the process 2900 may be performed byone or more other components of the server 110. The server 110 also ispresented as an exemplary device and the operations described herein maybe performed by any suitable device, such as one or more of theelectronic devices 102A-D. Further for explanatory purposes, the blocksof the process 2900 are described herein as occurring in serial, orlinearly. However, multiple blocks of the process 2900 may occur inparallel. In addition, the blocks of the process 2900 need not beperformed in the order shown and/or one or more of the blocks of theprocess 2900 need not be performed and/or can be replaced by otheroperations.

In the process 2900, the server 110 receives downlink condition reportsfrom the electronic devices 102A-C, respectively, each downlinkcondition report indicating a downlink channel bandwidth and/or adownlink channel condition corresponding to the respective electronicdevice (2902). In one or more implementations, the electronic devices102A-C may be participating in, or may be about to participate in, anaudio and/or video group communication session. The server 110 maydetermine uplink conditions with respect to the electronic devices102A-C (2904). For example, the server 110 may request that theelectronic devices 102A-C transmit a measurement packet to the server110 and/or otherwise participate in a protocol for determining uplinkchannel conditions.

The server 110 determines the appropriate quality tiers for the audioand/or video streams to be provided by each of the electronic devices102A-C (2906). The server 110 transmits indications of the determinedquality tiers to each of the respective electronic devices 102A-C(2908). In one or more implementations, the server 110 may transmit toone or more of the electronic devices 102A-C, such as the electronicdevice 102A, an indication of the quality tiers to be provided by theelectronic device 102A, as well as indications of the quality tiers tobe provided by the other electronic devices 102B-C. In this manner, theelectronic device 102A is made aware of the quality tiers that areavailable from the other electronic devices 102B-C.

The server 110 receives the audio and/or video streams from theelectronic devices 102A-C at the indicated quality tiers (2910), and theserver 110 relays one of the quality tiers of an audio and/or videostream from each of the electronic devices 102A-C to each of the otherelectronic devices 102A-C (2912).

FIG. 30 illustrates a flow diagram of an example process 3000 of themulti-device communication management system in accordance with one ormore implementations. For explanatory purposes, the process 3000 isprimarily described herein with reference to the server 110 and theelectronic devices 102A-B of FIG. 1. However, the process 3000 is notlimited to the server 110 and/or the electronic devices 102A-B of FIG.1, and one or more blocks (or operations) of the process 2900 may beperformed by one or more other components of the server 110 and/or theelectronic devices 102A-B. The server 110 and the electronic devices102A-B also are presented as exemplary devices and the operationsdescribed herein may be performed by any suitable device, such as one ormore of the other electronic devices 102C-D. Further for explanatorypurposes, the blocks of the process 3000 are described herein asoccurring in serial, or linearly. However, multiple blocks of theprocess 3000 may occur in parallel. In addition, the blocks of theprocess 3000 need not be performed in the order shown and/or one or moreof the blocks of the process 3000 need not be performed and/or can bereplaced by other operations.

Two or more of the electronic devices 102A-D, such as the electronicdevices 102A-B, may establish a group communication session utilizing afirst communication modality, such as one or more of messaging, audio(e.g. phone call), and/or video (e.g., an audio/video conference)(3002). In one or more implementations, the server 110 may facilitatethe electronic devices 102A-B with establishing the group communicationsession, e.g., in the manner described above with respect to FIGS. 6-14,such as to establish a logical and/or physical infrastructure for thegroup communication session (establishing participant identifiers,security keys, and the like). The two or more electronic devices 102A-Bmay engage in the group communication session utilizing the firstcommunication modality as well as utilizing at least one of: theparticipant identifiers or the security keys.

One or more of the electronic devices 102A-B, and/or the server 110, maydecide that the group communication session should utilize a secondcommunication modality, such as one or more of messaging, audio, and/orvideo that is not being utilized as the first communication modality(3004). For example, a user interacting with one or more of theelectronic devices 102A-B may provide an indication, such as by a verbalcommand, interfacing with a user interface, etc., indicating that amessaging conversation should transition to an audio/video conference.In one or more implementations, one or more of the electronic devices102A-B and/or the server 110 may determine based on one or more factors(e.g., available bandwidth, number of participants, etc.) that the groupcommunication session should utilize the second communication modality,and the electronic devices 102A-B and/or the server may adaptivelyinitiate the transition to utilizing the second communication modality,e.g., with or without confirmation from one or more of the users of theelectronic devices 102A-B.

The electronic devices 102A-B, e.g., with or without facilitation fromthe server 110, may transition the group communication session toutilize the second communication modality, such as to an audio/videoconference (3006). The group communication session may utilize thesecond communication modality in lieu of, or in addition to, the firstcommunication modality. In one or more implementations, the groupcommunication session may be transitioned to utilizing the secondcommunication modality without having to re-establish the infrastructurefor the group communication session. Thus, the electronic devices 102A-Bmay continue to use the participant identifiers, security keys, etc.,established for the group communication session utilizing the firstcommunication modality, e.g. messaging, while engaging in the groupcommunication session utilizing the second communication modality, e.g.,an audio/video conference.

FIG. 31 illustrates an electronic system 3100 with which one or moreimplementations of the subject technology may be implemented. Theelectronic system 3100 can be, and/or can be a part of, one or more ofthe electronic devices 102A-D and/or the server 110 shown in FIG. 1. Theelectronic system 3100 may include various types of computer readablemedia and interfaces for various other types of computer readable media.The electronic system 3100 includes a bus 3108, one or more processingunit(s) 3112, a system memory 3104 (and/or buffer), a ROM 3110, apermanent storage device 3102, an input device interface 3114, an outputdevice interface 3106, and one or more network interfaces 3116, orsubsets and variations thereof.

The bus 3108 collectively represents all system, peripheral, and chipsetbuses that communicatively connect the numerous internal devices of theelectronic system 3100. In one or more implementations, the bus 3108communicatively connects the one or more processing unit(s) 3112 withthe ROM 3110, the system memory 3104, and the permanent storage device3102. From these various memory units, the one or more processingunit(s) 3112 retrieves instructions to execute and data to process inorder to execute the processes of the subject disclosure. The one ormore processing unit(s) 3112 can be a single processor or a multi-coreprocessor in different implementations.

The ROM 3110 stores static data and instructions that are needed by theone or more processing unit(s) 3112 and other modules of the electronicsystem 3100. The permanent storage device 3102, on the other hand, maybe a read-and-write memory device. The permanent storage device 3102 maybe a non-volatile memory unit that stores instructions and data evenwhen the electronic system 3100 is off. In one or more implementations,a mass-storage device (such as a magnetic or optical disk and itscorresponding disk drive) may be used as the permanent storage device3102.

In one or more implementations, a removable storage device (such as afloppy disk, flash drive, and its corresponding disk drive) may be usedas the permanent storage device 3102. Like the permanent storage device3102, the system memory 3104 may be a read-and-write memory device.However, unlike the permanent storage device 3102, the system memory3104 may be a volatile read-and-write memory, such as random accessmemory. The system memory 3104 may store any of the instructions anddata that one or more processing unit(s) 3112 may need at runtime. Inone or more implementations, the processes of the subject disclosure arestored in the system memory 3104, the permanent storage device 3102,and/or the ROM 3110. From these various memory units, the one or moreprocessing unit(s) 3112 retrieves instructions to execute and data toprocess in order to execute the processes of one or moreimplementations.

The bus 3108 also connects to the input and output device interfaces3114 and 3106. The input device interface 3114 enables a user tocommunicate information and select commands to the electronic system3100. Input devices that may be used with the input device interface3114 may include, for example, alphanumeric keyboards and pointingdevices (also called “cursor control devices”). The output deviceinterface 3106 may enable, for example, the display of images generatedby electronic system 3100. Output devices that may be used with theoutput device interface 3106 may include, for example, printers anddisplay devices, such as a liquid crystal display (LCD), a lightemitting diode (LED) display, an organic light emitting diode (OLED)display, a flexible display, a flat panel display, a solid statedisplay, a projector, or any other device for outputting information.One or more implementations may include devices that function as bothinput and output devices, such as a touchscreen. In theseimplementations, feedback provided to the user can be any form ofsensory feedback, such as visual feedback, auditory feedback, or tactilefeedback; and input from the user can be received in any form, includingacoustic, speech, or tactile input.

Finally, as shown in FIG. 31, the bus 3108 also couples the electronicsystem 3100 to one or more networks and/or to one or more network nodes,such as a cellular base station or a wireless access point. In thismanner, the electronic system 3100 can be a part of a network ofcomputers (such as a LAN, a wide area network (“WAN”), or an Intranet,or a network of networks, such as the Internet. Any or all components ofthe electronic system 3100 can be used in conjunction with the subjectdisclosure.

Implementations within the scope of the present disclosure can bepartially or entirely realized using a tangible computer-readablestorage medium (or multiple tangible computer-readable storage media ofone or more types) encoding one or more instructions. The tangiblecomputer-readable storage medium also can be non-transitory in nature.

The computer-readable storage medium can be any storage medium that canbe read, written, or otherwise accessed by a general purpose or specialpurpose computing device, including any processing electronics and/orprocessing circuitry capable of executing instructions. For example,without limitation, the computer-readable medium can include anyvolatile semiconductor memory, such as RAM, DRAM, SRAM, T-RAM, Z-RAM,and TTRAM. The computer-readable medium also can include anynon-volatile semiconductor memory, such as ROM, PROM, EPROM, EEPROM,NVRAM, flash, nvSRAM, FeRAM, FeTRAM, MRAM, PRAM, CBRAM, SONOS, RRAM,NRAM, racetrack memory, FJG, and Millipede memory.

Further, the computer-readable storage medium can include anynon-semiconductor memory, such as optical disk storage, magnetic diskstorage, magnetic tape, other magnetic storage devices, or any othermedium capable of storing one or more instructions. In one or moreimplementations, the tangible computer-readable storage medium can bedirectly coupled to a computing device, while in other implementations,the tangible computer-readable storage medium can be indirectly coupledto a computing device, e.g., via one or more wired connections, one ormore wireless connections, or any combination thereof.

Instructions can be directly executable or can be used to developexecutable instructions. For example, instructions can be realized asexecutable or non-executable machine code or as instructions in ahigh-level language that can be compiled to produce executable ornon-executable machine code. Further, instructions also can be realizedas or can include data. Computer-executable instructions also can beorganized in any format, including routines, subroutines, programs, datastructures, objects, modules, applications, applets, functions, etc. Asrecognized by those of skill in the art, details including, but notlimited to, the number, structure, sequence, and organization ofinstructions can vary significantly without varying the underlyinglogic, function, processing, and output.

While the above discussion primarily refers to microprocessor ormulti-core processors that execute software, one or more implementationsare performed by one or more integrated circuits, such as ASICs orFPGAs. In one or more implementations, such integrated circuits executeinstructions that are stored on the circuit itself

Those of skill in the art would appreciate that the various illustrativeblocks, modules, elements, components, methods, and algorithms describedherein may be implemented as electronic hardware, computer software, orcombinations of both. To illustrate this interchangeability of hardwareand software, various illustrative blocks, modules, elements,components, methods, and algorithms have been described above generallyin terms of their functionality. Whether such functionality isimplemented as hardware or software depends upon the particularapplication and design constraints imposed on the overall system.Skilled artisans may implement the described functionality in varyingways for each particular application. Various components and blocks maybe arranged differently (e.g., arranged in a different order, orpartitioned in a different way) all without departing from the scope ofthe subject technology.

It is understood that any specific order or hierarchy of blocks in theprocesses disclosed is an illustration of example approaches. Based upondesign preferences, it is understood that the specific order orhierarchy of blocks in the processes may be rearranged, or that allillustrated blocks be performed. Any of the blocks may be performedsimultaneously. In one or more implementations, multitasking andparallel processing may be advantageous. Moreover, the separation ofvarious system components in the embodiments described above should notbe understood as requiring such separation in all embodiments, and itshould be understood that the described program components and systemscan generally be integrated together in a single software product orpackaged into multiple software products.

As used in this specification and any claims of this application, theterms “base station”, “receiver”, “computer”, “server”, “processor”, and“memory” all refer to electronic or other technological devices. Theseterms exclude people or groups of people. For the purposes of thespecification, the terms “display” or “displaying” means displaying onan electronic device.

As used herein, the phrase “at least one of” preceding a series ofitems, with the term “and” or “or” to separate any of the items,modifies the list as a whole, rather than each member of the list (i.e.,each item). The phrase “at least one of” does not require selection ofat least one of each item listed; rather, the phrase allows a meaningthat includes at least one of any one of the items, and/or at least oneof any combination of the items, and/or at least one of each of theitems. By way of example, the phrases “at least one of A, B, and C” or“at least one of A, B, or C” each refer to only A, only B, or only C;any combination of A, B, and C; and/or at least one of each of A, B, andC.

The predicate words “configured to”, “operable to”, and “programmed to”do not imply any particular tangible or intangible modification of asubject, but, rather, are intended to be used interchangeably. In one ormore implementations, a processor configured to monitor and control anoperation or a component may also mean the processor being programmed tomonitor and control the operation or the processor being operable tomonitor and control the operation. Likewise, a processor configured toexecute code can be construed as a processor programmed to execute codeor operable to execute code.

Phrases such as an aspect, the aspect, another aspect, some aspects, oneor more aspects, an implementation, the implementation, anotherimplementation, some implementations, one or more implementations, anembodiment, the embodiment, another embodiment, some embodiments, one ormore embodiments, a configuration, the configuration, anotherconfiguration, some configurations, one or more configurations, thesubject technology, the disclosure, the present disclosure, othervariations thereof and alike are for convenience and do not imply that adisclosure relating to such phrase(s) is essential to the subjecttechnology or that such disclosure applies to all configurations of thesubject technology. A disclosure relating to such phrase(s) may apply toall configurations, or one or more configurations. A disclosure relatingto such phrase(s) may provide one or more examples. A phrase such as anaspect or some aspects may refer to one or more aspects and vice versa,and this applies similarly to other foregoing phrases.

The word “exemplary” is used herein to mean “serving as an example,instance, or illustration”. Any embodiment described herein as“exemplary” or as an “example” is not necessarily to be construed aspreferred or advantageous over other embodiments. Furthermore, to theextent that the term “include”, “have”, or the like is used in thedescription or the claims, such term is intended to be inclusive in amanner similar to the term “comprise” as “comprise” is interpreted whenemployed as a transitional word in a claim.

All structural and functional equivalents to the elements of the variousaspects described throughout this disclosure that are known or latercome to be known to those of ordinary skill in the art are expresslyincorporated herein by reference and are intended to be encompassed bythe claims. Moreover, nothing disclosed herein is intended to bededicated to the public regardless of whether such disclosure isexplicitly recited in the claims. No claim element is to be construedunder the provisions of 35 U.S.C. § 112, sixth paragraph, unless theelement is expressly recited using the phrase “means for” or, in thecase of a method claim, the element is recited using the phrase “stepfor”.

The previous description is provided to enable any person skilled in theart to practice the various aspects described herein. Variousmodifications to these aspects will be readily apparent to those skilledin the art, and the generic principles defined herein may be applied toother aspects. Thus, the claims are not intended to be limited to theaspects shown herein, but are to be accorded the full scope consistentwith the language claims, wherein reference to an element in thesingular is not intended to mean “one and only one” unless specificallyso stated, but rather “one or more”. Unless specifically statedotherwise, the term “some” refers to one or more. Pronouns in themasculine (e.g., his) include the feminine and neuter gender (e.g., herand its) and vice versa. Headings and subheadings, if any, are used forconvenience only and do not limit the subject disclosure.

What is claimed is:
 1. A device, comprising: a memory; and at least oneprocessor configured to: transmit, to a first server, an allocationrequest requesting allocation of a group communication session with aplurality of devices; receive, from the first server, an allocationresponse in response to the allocation request, the allocation responseincluding credential information for the device to use to join the groupcommunication session; transmit, to a second server, an allocation bindrequest with the credential information to join the group communicationsession using the credential information; receive, from the secondserver, an allocation bind success response in response to theallocation bind request, the allocation bind success response indicatingthat the device has joined the group communication session; and transmita join notification to the plurality of devices via a third server toprovide notification that the device has joined the group communicationsession.
 2. The device of claim 1, wherein the first server comprises anallocator server, the second server comprises a relay server, and thethird server comprises a notification server.
 3. The device of claim 2,wherein the at least one processor is further configured to: receive another join notification from at least one of the plurality of devices,the other join notification notifying the device that the at least oneof the plurality of devices has joined the group communication session;transmit, to the relay server, an information request requestinginformation on the at least one of the plurality of devices; andreceive, from the relay server, an information response including theinformation on the at least one of the plurality of devices in responseto the information request.
 4. The device of claim 2, wherein the atleast one processor is further configured to: transmit, to the relayserver, an allocation unbind request to leave the group communicationsession; receive, from the relay server, an allocation unbind responsein response to the allocation unbind request to indicate that the devicehas left the group communication session; and transmit, via thenotification server, a leave notification to the plurality of devicesthat have joined the group communication session, the leave notificationnotifying the plurality of devices that the device has left the groupcommunication session.
 5. The device of claim 1, wherein the allocationrequest comprises a destination device list identifying the plurality ofdevices to receive allocation responses, the destination device listbeing contained in an array.
 6. The device of claim 1, wherein the atleast one processor is further configured to: receive a pre-key from atleast one of the plurality of devices; receive a master key from the atleast one of the plurality of devices, the master key being based on thepre-key; and communicate data with the at least one of the plurality ofdevices using the master key, wherein the communicated data is encryptedwith the master key.
 7. The device of claim 1, wherein at least two ofthe first, second, and third servers comprise a same server.
 8. Thedevice of claim 1, wherein at least two of the plurality of devices areassociated with a same user account.
 9. A non-transitorymachine-readable medium storing code that, when executed by a processor,causes the processor to perform operations, the code comprising: code toreceive, from a first device, an allocation request requesting anallocation of a group communication session; code to transmit anallocation response to the first device and one or more second devicesin response to the allocation request, the allocation response includingcredential information for the first device to use to join the groupcommunication session; code to receive, from the first device, anallocation bind request with the credential information to join thegroup communication session using the credential information; and codeto transmit, to the first device, an allocation bind success response inresponse to the allocation bind request, the allocation bind successresponse indicating that the first device has joined the groupcommunication session.
 10. The non-transitory machine-readable medium ofclaim 9, wherein the code further comprises: code to receive, from thefirst device, a join notification notifying that the first device hasjoined the group communication session; and code to forward the joinnotification to the one or more second devices.
 11. The non-transitorymachine-readable medium of claim 10, wherein the code further comprises:code to receive, from at least one of the one or more second devices, aninformation request requesting information on the first device; and codeto transmit, to the at least one of the one or more second devices, aninformation response including the information on the first device inresponse to the information request.
 12. The non-transitorymachine-readable medium of claim 10, wherein the code further comprises:code to receive, from the first device, an allocation unbind request toleave the group communication session; code to transmit, to the firstdevice, an allocation unbind response in response to the allocationunbind request to indicate that the first device has left the groupcommunication session; and code to transmit a leave notification to theone or more second devices that have joined the group communicationsession, the leave notification notifying that the first device has leftthe group communication session.
 13. A method comprising: transmitting,by a device, an allocation request requesting allocation of a groupcommunication session with a plurality of devices; receiving anallocation response in response to the allocation request, theallocation response including credential information for the device touse to join the group communication session; transmitting an allocationbind request with the credential information to join the groupcommunication session using the credential information; receiving anallocation bind success response in response to the allocation bindrequest, the allocation bind success response indicating that the devicehas joined the group communication session; and providing a joinnotification to the plurality of devices via an intermediary device tonotify that the device has joined the group communication session. 14.The method of claim 13, wherein the allocation response is received froman allocator server, the allocation bind success response is receivedfrom a relay server, and the intermediary device comprises anotification server.
 15. The method of claim 14, wherein the methodfurther comprises: receiving an other join notification from at leastone of the plurality of devices, the other join notification notifyingthe device that the at least one of the plurality of devices has joinedthe group communication session; transmitting, to the relay server, aninformation request requesting information on the at least one of theplurality of devices; and receiving, from the relay server, aninformation response including the information on the at least one ofthe plurality of devices in response to the information request.
 16. Themethod of claim 14, further comprising: transmitting, to the relayserver, an allocation unbind request to leave the group communicationsession; receive, from the relay server, an allocation unbind responsein response to the allocation unbind request to indicate that the devicehas left the group communication session; and transmit, via thenotification server, a leave notification to the plurality of devicesthat have joined the group communication session, the leave notificationnotifying the plurality of devices that the device has left the groupcommunication session.
 17. The method of claim 14, wherein at least twoof the allocator server, relay server, and notification server comprisea same server.
 18. The method of claim 13, wherein the allocationrequest comprises a destination device list identifying the plurality ofdevices to receive allocation responses, the destination device listbeing contained in an array.
 19. The method of claim 13, furthercomprising: receive a pre-key from at least one of the plurality ofdevices; receive a master key from the at least one of the plurality ofdevices, the master key being based on the pre-key; and communicate datawith the at least one of the plurality of devices using the master key,wherein the communicated data is encrypted with the master key.
 20. Themethod of claim 13, wherein at least two of the plurality of devices areassociated with a same user account.